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WHAT IS CLAIMED IS: 

1. A method comprising: 

generating a three dimensional model of an environment 
from range sensor information representing a height field for 
the environment; 

tracking orientation information of at least one image 
sensor in the environment with respect to the three dimensional 
model in real-time; 

projecting real-time video imagery information from the at 
least one image sensor onto the three dimensional model based 
on the tracked orientation information; and 

visualizing the three dimensional model with the projected 
real-time video imagery. 

2. The method of claim 1, wherein generating the three 
dimensional model comprises parametric fitting of geometric 
primitives to the range sensor information. 

3. The method of claim 2, wherein generating the three 
dimensional model further comprises: 

identifying a structure in the range sensor information; 

selecting a set of geometric primitives, including a 
superquadratic, to use in the parametric fitting with respect 
to the identified structure based on a shape of the structure. 
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4. The method of claim 2, wherein the at least one image 
sensor comprises multiple image sensors, and generating the 
three dimensional model further comprises refining the three 
dimensional model based on object surfaces mapped from images 
acguired by the image sensors. 

5. The method of claim 1, wherein generating the three 
dimensional model further comprises projecting and resampling 
points in the range sensor information onto a regular grid at a 
user-defined resolution to produce the height field. 

6. The method of claim 5, wherein generating the three 
dimensional model further comprises processing the height field 
using hole-filling and tessellation to generate a triangle mesh 
representation of the three dimensional model. 

7. The method of claim 2, wherein the range sensor 
information comprises light detection and ranging (LIDAR) 
sensor data from an active airborne laser sensor. 
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8. The method of claim 1, wherein tracking orientation 
information comprises tracking position and orientation 
information of the at least one image sensor by estimating a 
camera pose based at least in part on three dimensional 
parameters of point and line structures visible in the camera 
image, and projecting real-time video imagery information 
comprises projecting real-time video imagery information from 
the at least one image sensor onto the three dimensional model 
based on the tracked orientation information. 

9. The method of claim 8, wherein tracking position and 
orientation information of the at least one image sensor 
further comprises processing data from a tracking sensor system 
that integrates visual input, global navigational satellite 
system receiver input, and inertial orientation sensor input. 

10. The method of claim 1, wherein the at least one image 
sensor comprises multiple image sensors, and projecting the 
real-time video imagery information comprises projecting 
multiple video streams from the multiple image sensors onto the 
three dimensional model. 
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11. The method of claim 1, wherein projecting the real- 
time video imagery information comprises: 

generating a depth map image from a video sensor 
viewpoint; and 

projective texture mapping the real-time video imagery 
information onto the three dimensional model conditioned upon 
visibility as determined from the generated depth map image. 

12. The method of claim 11, wherein generating the depth 
map image and projective texture mapping the real-time video 
imagery information are performed using a one-pass approach on 
graphics hardware that supports SGI OpenGL extensions. 

13. The method of claim 1, wherein visualizing the three 
dimensional model comprises: 

video-projecting onto a display screen using a stereo 
video-projector; and 

coupling a rendering viewpoint to a user' s head position 
using data from a tracker. 
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14. An augmented virtual environment system comprising: 
a model construction component that generates a three 

dimensional model of an environment from range sensor 
information representing a height field for the environment; 

a dynamic fusion imagery projection component that 
projects real-time video imagery information from at least one 
image sensor onto the three dimensional model based on 
orientation information of the at least one image sensor 
tracked in the environment with respect to the three 
dimensional model in real-time; and 

a visualization sub-system that visualizes the three 
dimensional model with the projected real-time video imagery. 

15. The system of claim 14, wherein the at least one 
image sensor comprises multiple image sensors. 

16. The system of claim 15, wherein the real-time video 
imagery information comprises pre-recorded real-time video 
imagery information . 

17. The system of claim 15, wherein the dynamic fusion 
imagery projection component bases the real-time video imagery 
projection on a viewpoint separate from viewpoints associated 
with the multiple image sensors. 
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18. The system of claim 14, wherein the model 
construction component performs operations comprising: 

identifying a structure in the range sensor information; 

selecting a set of geometric primitives, including a 
superquadratic, based on a shape of the structure; and 

parametric fitting of the selected geometric primitives to 
the range sensor information. 

19. The system of claim 18, wherein the operations 
further comprise: 

projecting and resampling points in the range sensor 
information onto a regular grid at a user-defined resolution to 
produce the height field; and 

processing the height field using hole-filling and 
tessellation to generate a triangle mesh representation of the 
three dimensional model. 

20. The system of claim 14, wherein the at least one 
image sensor comprises multiple image sensors, and the system 
further comprises a model refinement component that refines the 
three dimensional model based on object surfaces mapped from 
images acquired by the image sensors. 
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21. The system of claim 14, further comprising a tracking 
sensor system that integrates visual input, global navigational 
satellite system receiver input, and inertial orientation 
sensor input to obtain position and orientation information of 
the at least one image sensor, and the dynamic fusion imagery 
projection component projects the real-time video imagery 
information based on the position and orientation information. 

22. The system of claim 14, wherein the visualization 
sub-system comprises the dynamic fusion imagery projection 
component and graphics hardware that supports SGI OpenGL 
extensions, and uses a one-pass approach on the graphics 
hardware to generate a depth map image from a video sensor 
viewpoint and projective texture map the real-time video 
imagery information to project the real-time video imagery 
information conditioned upon visibility as determined from the 
generated depth map image. 

23. The system of claim 14, wherein the visualization 
sub-system comprises: 

a stereo video-projector; and 
a tracker. 
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24. A method comprising: 

obtaining a three dimensional model of an environment; 

identifying in real time a region in motion with respect 
to a background image in real-time video imagery information 
from at least one image sensor having associated position and 
orientation information with respect to the three dimensional 
model, the background image comprising a single distribution 
background dynamically modeled from a time average of the real- 
time video imagery information; 

placing a surface that corresponds to the moving region in 
the three dimensional model; 

projecting the real-time video imagery information onto 
the three dimensional model, including the surface, based on 
the position and orientation information; and 

visualizing the three dimensional model with the projected 
real-time video imagery. 

25. The method of claim 24, wherein the surface comprises 
a two dimensional surface. 
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26. The method of claim 25, wherein placing the two 
dimensional surface comprises: 

casting a ray from an optical center, corresponding to the 
real-time video imagery information, to a bottom point of the 
moving region in an image plane in the three dimensional model; 
and 

determining a position, an orientation and a size of the 
two dimensional surface based on the ray, a ground plane in the 
three dimensional model, and the moving region. 

27. The method of claim 24, wherein identifying a region 
in motion in real time comprises: 

subtracting the background image from the real-time video 
imagery information; 

identifying a foreground object in the subtracted real- 
time video imagery information; 

validating the foreground object by correlation matching 
between identified objects in neighboring image frames; and 

outputting the validated foreground object. 

28. The method of claim 27, wherein identifying a 
foreground object comprises identifying the foreground object 
in the subtracted real-time video imagery information using a 
histogram-based threshold and a morphological noise filter. 
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29. The method of claim 28, wherein identifying a region 
in motion in real time further comprises estimating the 
background image by modeling the background image as a temporal 
pixel average of five recent image frames in the real-time 
video imagery information. 

30. The method of claim 24, further comprising tracking 
the position and orientation information of the at least one 
image sensor in the environment with respect to the three 
dimensional model in real-time. 

31. The method of claim 30, wherein obtaining a three 
dimensional model of an environment comprises generating the 
three dimensional model of the environment from range sensor 
information representing a height field for the environment. 
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32. An augmented virtual environment system comprising: 

an object detection and tracking component that identifies 
in real time, a region in motion with respect to a background 
image in real-time video imagery information from at least one 
image sensor having associated position and orientation 
information with respect to a three dimensional model, the 
background image comprising a single distribution background 
dynamically modeled from a time average of the real-time video 
imagery information, and places a surface that corresponds to 
the moving region with respect to the three dimensional model; 

a dynamic fusion imagery projection component that 
projects the real-time video imagery information onto the three 
dimensional model, including the surface, based on the position 
and orientation information; and 

a visualization sub-system that visualizes the three 
dimensional model with the projected real-time video imagery. 

33. The system of claim 32, wherein the surface comprises 
a two dimensional surface. 
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34. The system of claim 33, wherein the object detection 
and tracking component places the two dimensional surface by 
performing operations comprising: 

casting a ray from an optical center, corresponding to the 
real-time video imagery information, to a bottom point of the 
moving region in an image plane in the three dimensional model; 
and 

determining a position, an orientation and a size of the 
two dimensional surface based on the ray, a ground plane in the 
three dimensional model, and the moving region. 

35. The system of claim 32, wherein the object detection 
and tracking component identifies the moving region by 
performing operations comprising: 

subtracting the background image from the real-time video 
imagery information; 

identifying a foreground object in the subtracted real- 
time video imagery information; 

validating the foreground object by correlation matching 
between identified objects in neighboring image frames; and 

outputting the validated foreground object. 
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36. The system of claim 35, wherein identifying a 
foreground object comprises identifying the foreground object 
in the subtracted real-time video imagery information using a 
histogram-based threshold and a morphological noise filter. 

37. The system of claim 36, wherein identifying a region 
in motion in real time further comprises estimating the 
background image by modeling the background image as a temporal 
pixel average of five recent image frames in the real-time 
video imagery information. 

38. The system of claim 32, further comprising a tracking 
sensor system that integrates visual input, global navigational 
satellite system receiver input, and inertial orientation 
sensor input to obtain the position and the orientation 
information associated with the at least one image sensor in 
real time in conjunction with the real-time video imagery. 

39. The system of claim 38, further comprising a model 
construction component that generates the three dimensional 
model of an environment from range sensor information 
representing a height field for the environment. 
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40. A machine-readable medium embodying information 
indicative of instructions for causing one or more machines to 
perform operations comprising: 

generating a three dimensional model of an environment 
from range sensor information representing a height field for 
the environment; 

tracking orientation information of at least one image 
sensor in the environment with respect to the three dimensional 
model in real-time; 

projecting real-time video imagery information from the at 
least one image sensor onto the three dimensional model based 
on the tracked orientation information; and 

visualizing the three dimensional model with the projected 
real-time video imagery. 

41. The machine-readable medium of claim 40, wherein 
generating the three dimensional model comprises: 

identifying a structure in the range sensor information; 

selecting a set of geometric primitives, including a 
superquadratic, based on a shape of the structure; and 

parametric fitting of the selected geometric primitives to 
the range sensor information. 
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42. A machine-readable medium embodying information 
indicative of instructions for causing one or more machines to 
perform operations comprising: 

obtaining a three dimensional model of an environment; 

identifying in real time a region in motion with respect 
to a background image in real-time video imagery information 
from at least one image sensor having associated position and 
orientation information with respect to the three dimensional 
model, the background image comprising a single distribution 
background dynamically modeled from a time average of the real- 
time video imagery information; 

placing a surface that corresponds to the moving region in 
the three dimensional model; 

projecting the real-time video imagery information onto 
the three dimensional model, including the surface, based on 
the position and orientation information; and 

visualizing the three dimensional model with the projected 
real-time video imagery. 
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43. The machine-readable medium of claim 42, wherein 
placing the surface comprises: 

casting a ray from an optical center, corresponding to the 
real-time video imagery information, to a bottom point of the 
moving region in an image plane in the three dimensional model; 
and 

determining a position, an orientation and a size of the 
two dimensional surface based on the ray, a ground plane in the 
three dimensional model, and the moving region. 

44. The machine-readable medium of claim 42, wherein 
identifying a region in motion in real time comprises: 

subtracting the background image from the real-time video 
imagery information; 

identifying a foreground object in the subtracted real- 
time video imagery information; 

validating the foreground object by correlation matching 
between identified objects in neighboring image frames; and 

outputting the validated foreground object. 
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